SPE 163585 A New Approach to Load Balance for Parallel Compositional Simulation Based on Reservoir Model Over-decomposition

The quest for efficient and scalable parallel reservoir simulators has been evolving with the advancement of high performance computing architectures. Among the various challenges of efficiency and scalability, load imbalance is a major obstacle that has not been fully addressed and solved. The reasons that cause load imbalance in parallel reservoir simulation are both static and dynamic. Robust graph partitioning algorithms are capable of handling static load imbalance by decomposing the underlying reservoir geometry to distribute a roughly equal load to each processor. However, these loads determined by a static load balancer seldom remain unchanged as the simulation proceeds in time. This so called dynamic imbalance can be further exacerbated in parallel compositional simulations. The flash calculations for equations of state in complex compositional simulations not only can consume over half of the total execution time but also are difficult to balance merely by a static load balancer. The computational cost of flash calculations in each grid block heavily depends on the dynamic data such as pressure, temperature, and hydrocarbon composition. Thus, any static assignment of grid blocks may lead to dynamic load imbalance in unpredictable manners. A dynamic load balancer can often provide solutions for this difficulty. However, traditional techniques are inflexible and tedious to implement in legacy reservoir simulators. In this paper, we present a new approach to address dynamic load imbalance in parallel compositional simulation. It overdecomposes the reservoir model to assign each processor a bundle of subdomains. Processors treat these bundles of subdomains as virtual processes or user-level migratable threads which can be dynamically migrated across processors in the run-time system. This technique is shown to be capable of achieving better overlap between computation and communication for cache efficiency. We employ this approach in a legacy reservoir simulator and demonstrate reduction in the execution time of parallel compositional simulations while requiring minimal changes to the source code. Finally, it is shown that domain over-decomposition together with a load balancer can improve speedup from 29.27 to 62.38 on 64 physical processors for a realistic simulation problem. Introduction High performance computing including parallel computing plays a vital role in many areas of engineering, such as defense, energy and financial engineering. Nowadays, these devices are being widely used by domain application engineers and scientists to solve a variety of commercially and scientifically interesting computationally intensive problems. Many of the techniques utilized are associated with solving the discretized partial differential equations that describe the underlying physics. Reservoir simulation, which mimics or infers the behavior of fluid flow in a petroleum reservoir system through the use of mathematical models, is one of the methods that is widely used in petroleum upstream development and production. Reservoir simulator was born as an efficient tool for reservoir engineers to better understand and manage assets. However, like any numerical simulation tool, reservoir simulation is inherently computationally intensive and easily becomes inefficient if larger and larger grids or more components are necessary to describe accurately the complex phenomena occurring in the subsurface of the earth. Therefore, execution of reservoir simulation on parallel computers seems be the apparently feasible way to tackle the computational demand of reservoir simulation. Although running reservoir simulation in parallel sounds extremely attractive, developing an efficient parallel reservoir simulator is far more challenging than developing the underlying serial reservoir simulator. For decades there have remained many open problems associated with high performance computing and reservoir simulation. The quest for efficient and scalable parallel reservoir simulators has been evolving with the advancement of high performance computing architectures. The mainframes of decades ago quickly gave way to workstations, clusters, and finally